Rob Erdmann  0:02  
Yes, that's all so I guess I'll share my screen. This is not actually what I'm going to be talking about today, but it's a title slide from a keynote talk that I gave recently. So I'm, I'm, I'm Rob Erdmann. I'm a senior scientist at the Rijksmuseum and full professor at the University of Amsterdam split between conservation and illustration and physics, and before that I was at the University of Arizona, in material science and engineering and applied mathematics. I've been working on stitching and registration, among other things for about 13 years now. So I'll tell you a little bit about how we do these things at the Rijksmuseum. So, I write all my own software and so stitching and registration are the one of the most challenging problems we face. So the current system that we have basically does stitching and registration across arbitrary scales and arbitrary image types. So to begin with a computer uses a slightly modified version of a pre-trained convolutional neural network to find features across scales that allows us to make a initial pre-placement of the images, because everything we do involves placing high resolution images on a lower resolution groundtruth file to ensure that if there's any errors in the placement of one image, it doesn't propagate to the other images. Subsequent to that, regardless of which imaging types we have, so here is an example of a visible image and a radiograph, then a pair of convolutional neural networks learn a set of symmetric filters on a case by case basis that will pull out whatever common features exists between the two image types under consideration. So, in this case cracks are quite prominent feature. So this is the filtered version of the first image, this is the filtered version of the second image, and the network is designed so that these filters are learned, in order that the cross-correlation between this pair of filtered images has one cross-correlation that is nearly perfect, and all other relative offsets are terrible, so there's one right answer. Basically this then allows us to do this on a tile-wsie basis across the two image types to get, actually,

to get estimates of the local offset between the pairs of images, these offsets can can then be used with a Bayesian model to make a robust estimation of the homography to transform one image to the next. So after that is done, then we would have something like this for a pair of 100 megapixel images. This is still - so these are offsets in pixels -  so one and a half two pixels but that's still not good enough. So then we cascade down the scales to higher and higher resolution, learning separate filters at every stage, which then allows us to, to ultimately make a map function that would let us work one image into the next. This then works across arbitrary image types, so as an example of what we do for the Nightwatch, here we have five micron resolution image of the Nightwatch, hyperspectral images, which can be computed and registered across scales, radiographs, historical documents, for example photos of the historical slashing attack, macro X-ray fluorescence, and so on, sort of standard stuff, I guess, when it comes to image registration. And then we have this tool that allows us to to make viewers, and to make smooth fades and arbitrary combinations and so on, between the images. And then the last thing I would like to mention is that something that I think is quite important, but probably wouldn't otherwise be mentioned, is the need for semantic registration across across images even if they cannot be literally registered, so for example here is the Nightwatch on the left and a copy of the Nightwatch on the right. And so here is a an internet viewer that's rather than warping one image so that it aligns with the other is sort of dynamically registering them so that they align under the mouse, and then using neural networks we can finally imagine, semantic registration that does involve warping. So for example, here is the Nightwatch, this is the small copy of the Nightwatch by Gerrit Lunden's actually owned by the National Gallery of London but on long term loan to the RijksMuseum. So this is the best registration you can do without warping, but then with a semantic registration neural network you can turn this into this, which then is beautifully semantically aligned with the original painting, so of course this then broadens the possibilities before comparing objects within a collection. Or for example tapestry and a drawing. So that's all I have.